Army  Research  Laboratory 


A  Summary  of  Crew  Workload  and  Situational  Awareness 
Ratings  for  U.S.  Army  Aviation  Aircraft 


by  Jamison  S.  Hicks,  David  B.  Durbin,  Anthony  W.  Morris,  and 

Brad  M.  Davis 


ARL-TR-6955 


June  2014 


Approved  for  public  release;  distribution  is  unlimited. 


NOTICES 

Disclaimers 

The  findings  in  this  report  are  not  to  be  construed  as  an  official  Department  of  the  Army  position  unless 
so  designated  by  other  authorized  documents. 

Citation  of  manufacturer’s  or  trade  names  does  not  constitute  an  official  endorsement  or  approval  of  the 
use  thereof. 


Destroy  this  report  when  it  is  no  longer  needed.  Do  not  return  it  to  the  originator. 


Army  Research  Laboratory 

Aberdeen  Proving  Ground,  MD  21005-5425 


ARL-TR-6955 


June  2014 


A  Summary  of  Crew  Workload  and  Situational  Awareness 
Ratings  for  U.  S.  Army  Aviation  Aircraft 

Jamison  S.  Hicks,  David  B.  Durbin,  Anthony  W.  Morris,  and  Brad  M.  Davis 
Human  Research  and  Engineering  Directorate,  ARL 


Approved  for  public  release;  distribution  is  unlimited. 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering 
and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  the  burden,  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson 
Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to 
comply  with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 

The  U.S.  Army  Research  Laboratory  Human  Research  and  Engineering  Directorate  (ARL/HRED)  assesses  crewstation  design 
for  new  and  upgraded  U.S.  Army  Aviation  aircraft  during  simulations  and  operational  testing.  The  assessments  are  conducted 
to  identify  and  eliminate  human  factors  design  problems.  ARL/HRED  evaluates  pilot  workload  and  situational  awareness  (SA) 
during  simulations  and  operational  testing  to  assess  crewstation  design.  To  date,  more  than  12,000  pilot  workload  ratings  and 
3000  pilot  SA  ratings  have  been  collected  by  ARL/HRED  for  Army  aircraft.  This  report  summarizes  the  Bedford  Workload 
Rating  Scale,  Situation  Awareness  Rating  Technique,  and  China  Lake  Situational  Awareness  scale  ratings  collected  during 
simulations  and  operational  testing  to  assess  crewstation  design. 

15.  SUBJECT  TERMS 

workload,  situation  awareness,  Apache  Longbow,  UH60M,  CH47H,  simulation,  U.S.  Army  Aviation 

17.  LIMITATION  18.  NUMBER 
OF  ABSTRACT  OF  PAGES 

UU _ 50 

Standard  Form  298  (Rev.  8/98) 

Prescribed  by  ANSI  Std.  Z39.18 


19a.  NAME  OF  RESPONSIBLE  PERSON 

Jamison  Hicks 

19b.  TELEPHONE  NUMBER  (Include  area  code) 

334-255-2203 


16.  SECURITY  CLASSIFICATION  OF: 


a.  REPORT 

b.  ABSTRACT 

C.  THIS  PAGE 

Unclassified 

Unclassified 

Unclassified 

1.  REPORT  DATE  (DD-MM-YYYY) 

2.  REPORT  TYPE 

June  2014 

Final 

4.  TITLE  AND  SUBTITLE 

A  Summary  of  Crew  Workload  and  Situational  Awareness  Ratings  for  U.S.  Army 
Aviation  Aircraft 


6.  AUTHOR(S) 

Jamison  S.  Hicks,  David  B.  Durbin,  Anthony  W.  Morris,  and  Brad  M.  Davis 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Laboratory 

ATTN:  RDRL-HRM-DJ 

Aberdeen  Proving  Ground,  MD  21005-5425 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


3.  DATES  COVERED  (From  -  To) 
September-December  2013 
5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 


5c  PROGRAM  ELEMENT  NUMBER 


5d.  PROJECT  NUMBER 

62716AH70 

5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

ARL-TR-6955 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


11 


Contents 


List  of  Figures  v 

List  of  Tables  v 

1.  Assessment  of  Crew  Workload  and  Situational  Awareness  1 

1.1  Overview . 1 

1.2  Workload . 1 

1.2.1  Bedford  Workload  Rating  Scale . 2 

1.3  Situational  Awareness . 4 

1.4  Situation  Awareness  Rating  Technique  and  China  Lake  Situational  Awareness  Scale. .5 

1.5  Situation  Awareness  of  Battlefield  Elements . 7 

1.6  Additional  Metrics . 8 

1.6.1  PCI  Evaluation . 9 

1.6.2  Subject  Matter  Experts . 9 

1.6.3  Pilot  Interview  s . 9 

2.  Crewstation  Assessment  During  Simulation  and  Operational  Testing  9 

2.1  Crewstation  Assessment  Methodology . 9 

2.2  Simulation  and  Operational  Testing . 10 

2.3  Visual  Workload . 12 

3.  Workload  and  SA  Ratings  13 

3.1  Bedford  Workload  Rating  Scale  Ratings . 13 

3.2  SART  and  CLSA  Ratings . 15 

4.  Summary  16 

5.  References  18 

Appendix  A.  Bedford  Workload  Rating  Scale  and  Questionnaire  19 


Appendix  B.  SART  and  CLSA  Rating  Scales 


23 


Appendix  C.  PCI  Questionnaire  Example  29 

Appendix  D.  SME  Questionnaire  35 

List  of  Symbols,  Abbreviations,  and  Acronyms  39 

Distribution  List  41 


IV 


List  of  Figures 


Figure  1.  Bedford  Workload  Rating  Scale . 3 

Figure  2.  Endsley  model  of  situation  awareness  in  dynamic  decisionmaking . 5 

Figure  3.  OH-58F  crewstation  simulator . 12 

Figure  4.  AH-64D  Apache  Longbow  crewstation  simulator . 12 

Figure  5.  Eye  tracker  data  collected  from  AH-64D  Apache . 13 


List  of  Tables 


Table  1.  Sample  flight  and  mission  task  list . 4 

Table  2.  SART  dimensions . 6 

Table  3.  CLSA  scale . 7 

Table  4.  Battlefield  elements . 8 

Table  5.  U.S.  Army  aircraft,  associated  simulator,  and  assessment  or  test . 11 

Table  6.  Overall  workload  averages . 14 

Table  7.  Overall  SART  and  CLSA  averages . 16 


v 


Intentionally  lelt  blank. 


vi 


1.  Assessment  of  Crew  Workload  and  Situational  Awareness 


1.1  Overview 

Effective  aircrew  performance  is  critical  to  mission  success.  Crewstations  that  are  designed  to 
augment  the  cognitive  and  physical  abilities  of  aircrews  help  minimize  pilot  workload,  enhance 
situational  awareness  (SA),  and  contribute  to  successful  mission  performance.  It  is  vital  that 
crewstations  be  assessed  early  and  often  during  development  to  ensure  optimal  design. 

The  U.S.  Army  Research  Laboratory’s  Human  Research  and  Engineering  Directorate 
(ARL/HRED)  assesses  crewstation  design  for  new  and  upgraded  Army  Aviation  aircraft  during 
simulations  and  operational  testing.  The  assessments  are  conducted  to  identify  and  eliminate 
human  factors  design  problems.  ARL/HRED  evaluates  pilot  workload  and  SA  during 
simulations  and  operational  testing  to  assess  crewstation  design.  To  date,  more  than  12,000  pilot 
workload  ratings  and  3000  pilot  SA  ratings  have  been  collected  by  ARL/HRED  for  Army 
aircraft.  This  report  summarizes  the  Bedford  Workload  Rating  Scale  (BWRS),  Situation 
Awareness  Rating  Technique  (SART),  and  China  Lake  Situational  Awareness  (CLSA)  scale 
ratings  collected  during  simulations  and  operational  testing  to  assess  crewstation  design. 

1.2  Workload 

There  are  several  definitions  of  pilot  workload.  A  useful  definition  is  “the  integrated  mental  and 
physical  effort  required  to  satisfy  the  perceived  demands  of  a  specified  flight  task”  (Roscoe, 
1985).  It  is  important  to  assess  pilot  workload  because  mission  accomplishment  is  related  to  the 
mental  and  physical  ability  of  the  crew  to  effectively  perform  their  flight  and  mission  tasks.  If 
one  or  both  pilots  experience  excessively  high  workload  while  performing  flight  and  mission 
tasks,  the  tasks  may  be  performed  ineffectively  or  abandoned.  In  general,  workload  assessment 
techniques  are  used  to  answer  the  following  questions  (Eisen,  1987): 

•  Does  the  operator  have  the  capability  to  perform  the  required  tasks? 

•  Does  the  operator  have  enough  spare  capacity  to  take  on  additional  tasks? 

•  Does  the  operator  have  enough  spare  capacity  to  cope  with  emergency  situations? 

•  Can  the  task  or  equipment  be  altered  to  increase  the  amount  of  spare  capacity? 

•  Can  the  task  or  equipment  be  altered  to  increase/decrease  the  amount  of  mental  workload? 

•  How  does  the  workload  of  a  new  system  compare  with  the  old  system? 

To  assess  whether  the  pilots  are  task-overloaded  during  the  missions,  the  level  of  workload  for 
each  pilot  must  be  evaluated. 
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1.2.1  Bedford  Workload  Rating  Scale 

The  BWRS  has  been  used  extensively  by  the  military,  civil,  and  commercial  aviation 
communities  for  pilot  workload  estimation  (Roscoe  and  Ellis,  1990).  It  requires  pilots  to  rate  the 
level  of  workload  associated  with  a  task  based  on  the  amount  of  spare  capacity  they  feel  they 
have  to  perform  additional  tasks.  Spare  workload  capacity  is  an  important  commodity  for  pilots 
because  they  are  often  required  to  perform  several  tasks  concurrently.  For  example,  pilots  often 
perform  navigation  tasks,  communicate  via  multiple  radios,  monitor  aircraft  systems,  and  assist 
the  pilot  on  the  controls  with  flight  tasks  (e.g.,  maintain  airspace  surveillance)  within  the  same 
time  interval.  Mission  performance  is  reduced  if  pilots  are  task-saturated  and  have  little  or  no 
spare  capacity  to  perform  other  tasks. 

After  each  mission,  pilots  rate  flight  and  mission  tasks  using  the  BWRS  scale  (figure  1).  Roscoe 
and  Ellis  (1990)  describe  the  BWRS  and  explain  its  use  in  assessing  pilot  workload.  The  pilot 
starts  the  decisionmaking  process  at  the  bottom  left  corner  of  the  decision  tree,  which  consists  of 
three  questions  requiring  yes  or  no  answers,  in  order  to  proceed  to  the  descriptions  of  different 
levels  of  workload.  The  descriptors  denote  increasing  levels  of  workload  associated  with  ratings 
1-10.  The  flight  and  mission  tasks  should  be  well  defined  and  the  workload  being  assessed 
should  be  related  to  the  execution  of  the  primary  task;  any  additional  tasks  (e.g.,  monitoring 
communications)  must  be  included  as  part  of  spare  capacity.  Once  the  pilot  settles  on  a  workload 
rating  using  the  decision  tree,  the  rating  is  made  for  the  task  for  the  mission  or  mission  segment. 
A  sample  task  list  used  by  ARL/HRED  is  shown  in  table  1  and  a  full  task  list  is  in  appendix  A. 
For  several  aircraft,  ARL/HRED  wrote  the  workload  requirement  that  operation  of  the  aircraft 
must  not  result  in  aggregate  pilot  BWRS  task  and  mission  ratings  of  more  than  5. 0-6.0.  Flight 
and  mission  tasks  that  are  rated  5.0  or  above  are  evaluated  to  determine  if  crewstation  design 
problems  contributed  to  higher  workload  ratings. 
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Workload  Description 


Rating 


Workload  insignificant 

1 

Workload  low 

2 

Enough  spare  capacity  for  all 
desirable  additional  tasks 

3 

Is  it  possible  to 
complete  the  task? 


Task  abandoned.  Pilot  unable  to 
apply  sufficient  effort 


Pilot  Decisions 


Figure  1.  Bedford  Workload  Rating  Scale. 
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Table  1.  Sample  flight  and  mission  task  list. 


Task 

No. 

Flight  and  Mission  Tasks 

Workload  Rating 

1026 

Maintain  airspace  surveillance 

1028 

Perform  hover  power  check 

1030 

Perform  hover  out-of-ground-effect  check 

1032 

Perform  radio  communication  procedures 

1038 

Perform  hovering  flight 

1.3  Situational  Awareness 

SA  can  be  defined  as  the  pilot’s  mental  model  of  the  current  state  of  the  flight  and  mission 
environment.  A  more  fonnal  definition  is  “The  perception  of  the  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of 
their  status  in  the  near  future”  (Endsley,  1988).  The  Endsley  model  of  SA  in  dynamic 
decisionmaking  can  be  seen  in  figure  2  (Endsley,  1995).  The  SA  model  encompasses  three  levels 
of  SA.  Level  1  includes  the  perception  of  elements  in  the  current  situation,  level  2  is  the 
comprehension  of  the  current  situation,  and  level  3  is  the  projection  of  future  status.  The 
understanding  of  how  these  three  levels  of  SA  form  an  overview  of  pilot  understanding  is 
important.  Each  mission  segment  and  stimulus  may  call  for  different  levels  of  SA.  Maintaining  a 
single  level  of  SA  would  not  be  appropriate  for  all  scenarios,  especially  when  evaluated  with 
workload.  For  example,  a  requirement  for  level  3  SA  in  all  scenarios  would  burden  pilots  with 
workload  requirements  to  maintain  and  identify  level  3  SA  for  all  variables.  Ideally,  SA  is 
transferred  at  different  levels  among  relative  variable  occurrences,  and  an  overall  concept  of  the 
mission  and  aircraft  parameters  is  established  that  is  constantly  revised  throughout  the  mission 
and  based  on  pilot  actions.  It  is  important  to  assess  SA  because  of  its  potential  to  directly  impact 
pilot  and  aircraft  performance.  Good  SA  should  increase  the  probability  of  good  decisionmaking 
and  performance  by  aircrews  when  conducting  flight  and  mission  tasks. 
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Figure  2.  Endsley  model  of  situation  awareness  in  dynamic  decisionmaking. 


1.4  Situation  Awareness  Rating  Technique  and  China  Lake  Situational  Awareness  Scale 

The  SART  (appendix  B)  is  a  multidimensional  rating  scale  for  pilots  to  report  their  perceived 
SA.  SART  uses  10  dimensions  (table  2)  to  measure  operator  SA:  familiarity  of  the  situation, 
focusing  of  attention,  information  quantity,  information  quality,  instability  of  the  situation, 
concentration  of  attention,  complexity  of  the  situation,  variability  of  the  situation,  arousal,  and 
spare  mental  capacity.  SART  is  administered  after  each  mission  by  ARL/HRED  personnel  and 
involves  the  participant  rating  each  dimension  on  a  seven-point  rating  scale  (1  =  low,  7  =  high) 
to  gain  a  subjective  measure  of  SA  (Salmon  et  al.,  2006).  SART  was  developed  as  an  evaluation 
tool  for  the  design  of  aircrew  systems  (Taylor,  1989)  and  assesses  three  components  of  SA: 
understanding,  supply,  and  demand.  These  components  are  subcategories  that  contain  the  10 
dimensions.  Taylor  proposed  that  SA  depends  on  the  pilot’s  understanding  (U)  (e.g.,  quality  of 
information  they  receive)  and  the  difference  between  the  demand  (D)  on  the  pilot’s  resources 
(e.g.,  complexity  of  mission)  and  the  pilot’s  supply  (S)  (e.g.,  ability  to  concentrate).  When  D 
exceeds  S,  there  is  a  negative  effect  on  U  and  an  overall  reduction  of  SA.  The  formula 
SA  =  U  -  (D  -  S)  is  used  to  derive  the  overall  SART  score.  The  SART  is  one  of  the  most 
thoroughly  tested  rating  scales  for  estimating  SA  (Endsley,  2000). 
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Table  2.  SART  dimensions. 


Situation  Awareness  Rating  Technique  (SART) 

Demand 

Instability  of 
situation 

Likeliness  of  situation  to  change  suddenly 

V  ariability  of 
situation 

Number  of  variables  that  require  your  attention 

Complexity  of 
situation 

Degree  of  complication  (number  of  closely  connected  parts)  of  the  situation 

Supply 

Arousal 

Degree  to  which  you  are  ready  for  activity;  ability  to  anticipate  and  keep  up  with  the  flow  of 
events 

Spare  mental 
capacity 

Amount  of  mental  ability  available  to  apply  to  new  tasks 

Concentration 

Degree  to  which  your  thoughts  are  brought  to  bear  on  the  situation;  degree  to  which  you 
focused  on  important  elements  and  events 

Division  of 
attention 

Ability  to  divide  your  attention  among  several  key  issues  during  the  mission;  ability  to 
concern  yourself  with  many  aspects  of  current  and  future  events  simultaneously 

Understanding 

Information 

quantity 

Amount  of  knowledge  received  and  understood 

Information 

quality 

Degree  of  goodness  or  value  of  knowledge  communicated 

Familiarity 

Degree  of  acquaintance  with  the  situation 

ARL/HRED  personnel  also  use  the  CLSA  (table  3),  which  is  a  five-point  SA  scale  that  pilots  use 
to  rate  their  perceived  level  of  SA  for  the  mission  or  mission  segment.  The  scale  relates  to 
general  knowledge  of  aircraft  systems,  the  tactical  environment,  and  the  mission  objectives  as 
well  as  the  ability  to  accommodate  trends  and  the  level  of  task  shedding  due  to  high  workload. 
The  CLSA  is  considered  to  have  positive  attributes  of  high  face  validity  or  “common  sense,” 
clear  criteria,  ease  of  use,  and  ease  of  analysis. 
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Table  3.  CLSA  scale. 


CLSA  Scale  Value 

Content 

Very  good  -  1 

Full  knowledge  of  aircraft  energy  state,  and  tactical  environment/mission 

Full  ability  to  anticipate/accommodate  trends 

Good  -  2 

Full  knowledge  of  aircraft  energy  state,  tactical  environment  and  mission 

Partial  ability  to  anticipate/accommodate  trends 

No  task  shedding 

Adequate  -  3 

Full  knowledge  of  aircraft  energy  state,  tactical  environment  and  mission 

Saturated  ability  to  anticipate  and  accommodate  trends 

Some  shedding  of  minor  tasks 

Poor  -  4 

Fair  knowledge  of  aircraft  energy  state,  tactical  environment,  and  mission 

Saturated  ability  to  anticipate  and  accommodate  trends 

Shedding  of  all  minor  tasks  as  well  as  many  not  essential  to  flight  safety/mission 
effectiveness 

Very  poor  -  5 

Minimal  knowledge  of  aircraft  energy  state,  tactical  environment,  and  mission 

Oversaturated  ability  to  anticipate  and  accommodate  trends 

Shedding  of  all  tasks  not  essential  to  flight  safety  and  mission  effectiveness 

1.5  Situation  Awareness  of  Battlefield  Elements 

Assessing  crew  S  A  of  battlefield  elements  (table  4)  is  important  to  evaluate  their  comprehension 
of  the  elements  (e.g.,  threat  vehicles)  during  simulation  and  testing.  Pilots  provide  ratings  for 
each  element,  and  these  ratings  are  used  to  determine  whether  the  elements  are  identified  and 
understood  throughout  the  mission. 
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Table  4.  Battlefield  elements. 


Battlefield 

Elements 

Very  High 
Level  of 
Situation 
Awareness 

Fairly  High 
Level  of 
Situation 
Awareness 

Borderline 

Fairly  Low 
Level  of 
Situation 
Awareness 

Very  Low 
Level  of 
Situation 
Awareness 

Location  of 
enemy  units 

Location  of 
friendly  units 

Location  of 
noncombatants 
(e.g.,  civilians) 

Location  of  my 
aircraft  during 
missions 

Location  of 
other  aircraft 
in  my  flight 

Location  of 
cultural 
features  (e.g., 
bridges) 

Route 

information 
(e.g.,  air 
control  points) 

Status  of  my 
aircraft 
systems  (e.g., 
fuel 

consumption) 

1.6  Additional  Metrics 

In  addition  to  the  BWRS,  SART,  and  CLSA  data,  other  metrics  are  used  to  evaluate  the  overall 
crew  workload  and  SA  during  missions.  These  sources  include  a  Pilot-Crewstation  Interface 
(PCI)  survey  distributed  at  the  end  of  all  missions,  subject  matter  expert  (SME)  ratings,  and  pilot 
interviews. 
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1.6.1  PCI  Evaluation 


PCI  evaluations  are  used  to  examine  the  interaction  between  the  pilots  and  the  crewstation 
interface.  The  PCI  impacts  crew  workload  and  SA  during  a  mission.  A  PCI  that  is  designed  to 
augment  the  cognitive  and  physical  abilities  of  crews  will  minimize  workload,  enhance  SA,  and 
contribute  to  successful  mission  performance.  To  assess  the  PCI,  the  pilots  report  any  problems 
that  contributed  to  high  workload  and  low  SA  at  the  end  of  each  mission.  They  also  complete  a 
lengthy  questionnaire  (appendix  C)  at  the  end  of  their  final  mission.  The  questionnaire  addresses 
usability  characteristics  of  the  PCI  (e.g.,  software  interface,  control  reach,  and  button  presses). 

1.6.2  Subject  Matter  Experts 

SMEs  observe  the  missions  and  rate  pilot  workload  and  SA,  mission  success,  and  levels  of  crew 
coordination  (appendix  D)  that  they  observe  during  the  missions.  A  SME  is  typically  an 
experienced  pilot  that  has  in-depth  knowledge  of  the  aircraft  and  crewstation  being  assessed.  The 
ratings  provided  by  the  SME  are  compared  with  the  corresponding  pilot  ratings  to  identify  any 
significant  anomalies  in  perceived  levels  of  workload  or  SA  while  interacting  with  the 
crewstation. 

1.6.3  Pilot  Interviews 

Pilots  are  formally  interviewed  about  their  performance  during  after-action  reviews  in  which  the 
mission  events  and  goal  outcomes  are  discussed.  Pilots  are  also  interviewed  informally  by 
ARL/HRED  researchers  throughout  the  test  process  to  gain  insights  into  procedures  and  to 
capture  any  additional  comments  or  perceptions  of  workload  and  SA  experienced  during  the 
missions.  Pilots  also  complete  a  form  providing  recommendations  for  improvements  to  the 
crewstation.  Recommended  design  improvements  made  by  the  pilots  are  addressed  in  future 
design  analyses  and  incorporated  into  software  and  hardware  modifications  (within  funding 
constraints). 


2.  Crewstation  Assessment  During  Simulation  and  Operational  Testing 


2.1  Crewstation  Assessment  Methodology 

ARL/HRED  developed  a  methodology  to  assess  crewstation  design.  The  methodology  includes: 
anthropometric  modeling;  human  factors  evaluation  during  simulation  and  operational  testing  to 
assess  pilot  workload,  SA,  crew  coordination,  PCI,  and  anthropometric  accommodation;  and  use 
of  a  head  and  eye  tracker  to  assess  visual  gaze  and  dwell  times.  This  methodology  has  been  used 
to  help  develop  all  modernized  U.S.  Army  Aviation  systems  including  the  AH-64D/E  Apache, 
UH-60M  Blackhawk,  CH-47F  Chinook,  OH-58F  Kiowa,  and  UH-72A  Lakota.  The  methodology 
was  also  used  to  develop  aircraft  that  are  no  longer  being  developed  by  the  Army,  including  the 
C-27J  Spartan,  RAH-66  Comanche,  and  Armed  Reconnaissance  Helicopter  (ARH).  Data 
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collected  using  this  methodology  are  analyzed  and  used  to  construct  a  comprehensive  profile  of 
the  PCI  characteristics  and  impact  on  crew  workload  and  SA.  Additional  information  and 
summary  results  about  each  of  the  steps  in  the  methodology  can  be  found  in  a  report  (Hicks  and 
Durbin,  2013)  that  explains  the  crewstation  assessment  process. 

2.2  Simulation  and  Operational  Testing 

The  simulators  used  by  ARL/HRED  for  crewstation  design  assessments  are  engineering 
simulators.  The  engineering  simulators  represent  the  intended  production  design  and  provide  a 
platform  for  developing  and  assessing  crewstation  design,  evaluating  pilot  performance,  and 
assessing  crew  workload,  SA,  and  crew  coordination.  Pilots  with  broad  levels  of  experience 
(e.g.,  500-4000  flight  hours)  typically  participate  in  the  simulation  events.  This  wide  range  of 
experience  provides  ARL/HRED  researchers  a  broad  perspective  on  the  design  of  each 
crewstation.  Pilots  use  the  simulators  to  conduct  operationally  relevant  missions  and  tasks 
(e.g.,  zone  reconnaissance,  call  for  fire,  troop  transport).  The  simulators  are  also  used  to  help 
pilots  develop  tactics,  techniques  and  procedures  and  provide  limited  training  for  pilots  prior  to 
operational  testing  in  the  aircraft.  Results  of  the  assessments  are  provided  by  ARL/HRED  to  the 
aircraft  program  managers,  Training  and  Doctrine  Command  Capability  Managers,  Army  Test 
and  Evaluation  Command,  Aviation  and  Missile  Research,  Development  and  Engineering 
Center,  and  defense  contractors. 

Simulators  used  by  ARL/HRED  include  the  OH-58F  Kiowa,  AH-64D  Apache  Longbow  Risk 
and  Cost  Reduction  Simulator  (RACRS),  UH-60M  Blackhawk  Helicopter  Engineering  and 
Analysis  Cockpit  (BHEAC)  -  Battlefield  Highly  Immersive  Virtual  Environment  1  (BHIVE  1) 
and  Systems  Integration  Laboratory  (SIL)  simulators,  CH-47F  Chinook  Helicopter  Engineering 
and  Analysis  Cockpit  (CHEAC)  -  BHIVE  2  simulator,  ARH  simulator  -  BHIVE  2,  and  the 
RAH-66  Comanche  Engineering  Development  Simulator  (EDS)  and  Comanche  Portable  Cockpit 
(CPC).  The  simulators  contained  the  hardware  and  software  that  emulated  the  controls,  flight 
characteristics,  out-the-window  display,  and  functionality  of  the  aircraft.  The  simulator 
crewstations  replicated  the  corresponding  crewstation  in  the  actual  aircraft,  allowing  each  pilot  to 
perform  realistic  flight  and  mission  tasks.  Table  5  lists  the  aircraft,  associated  simulator,  and 
assessment  or  operational  test  that  was  conducted. 

Operational  tests  are  conducted  to  verify  design  requirements  and  ensure  crewstation  designs  are 
ready  for  fielding.  ARL/HRED  helps  conduct  operational  tests  and  typically  collects  the  same 
data  as  was  collected  during  the  simulations.  This  provides  a  progressive  assessment  of  pilot 
performance  and  crewstation  design.  Results  from  the  operational  test  are  compared  with  the 
simulations  to  ensure  improvements  have  been  made  to  the  crewstation  design  and  to  identify 
any  new  human  factors  design  problems. 
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Table  5.  U.S.  Army  aircraft,  associated  simulator,  and  assessment  or  test. 


Aircraft 

Simulator 

Assessment/Test 

AH-64D 

RACRS 

Unmanned  Aircraft  System 
(UAS)  Teaming  for  Integrated 
and  Federated  Systems 

AH-64E 

— 

Limited  User  Test  (LUT)  and 
Initial  Operational  Test  (IOT) 

ARH 

BHIVE  2 

Common  Aviation  Architecture 
System  (CAAS)  Assessment 

LUT 

CH-47F 

BHIVE  2 

CAAS  Assessment 

Horizontal  Situation 
Display-Hover  (HSDH) 

2-D  HUD  Improvements 

Air  Soldier  System 

3-D  Conformal  Symbology 

C-27J 

— 

Multi-Service  Operational  Test 
(MOT) 

OH-58F 

BHIVE  2 

Human  Factors  Engineering 
(HFE)  #1,  2,  3  Design 
Assessment 

RAH -66 

CPC,  EDS 

Force  Development  Test  and 
Experimentation  (FDT&E)  1 

UH-60L 

BHIVE  1,  BHEAC,  SIL 

Air  Soldier  System 
Multi-Modal  Cueing 

Pre-MS  B 

Air  Soldier  System 

3-D  Conformal  Symbology 

UH-60M 

BHIVE  1,  BHEAC,  SIL 

Early  User  Demonstration 
(EUD) 

Limited  Early  User 
Evaluation  (LEUE) 

LUT 

Required  Navigation 
Performance/ Area  Navigation 
(RNP/RNAV)  EUD 

Tactical  Airspace  Integration 
(TAIS)  1  &  2 

Mission  Information 
Management  (MIM)  1  &  2 
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Examples  of  simulators  that  ARL/HRED  uses  to  assess  pilot  workload  and  SA  include  the 
OH-58F  simulator  (figure  3)  and  the  AH-64D  Apache  simulator  (figure  4). 


Figure  3.  OH-58F  crewstation  simulator. 


Figure  4.  AH-64D  Apache  Longbow  crewstation  simulator. 

2.3  Visual  Workload 

To  help  assess  pilot  workload,  SA,  and  overall  crewstation  design,  pilots  wear  an  eye  tracker  to 
record  visual  gaze  and  dwell  times  during  missions  conducted  in  the  simulators  equipped  with 
head  trackers.  Recording  visual  gaze  and  dwell  times  can  identify  improvements  that  need  to  be 
made  to  crewstation  design  by  identifying  mission  or  task  segments  that  require  high  visual 
workload.  For  example,  if  pilots  spend  an  excessive  amount  of  time  viewing  the  crewstation 
displays,  this  can  indicate  that  the  displays  contain  information  that  requires  too  many  steps  (e.g. 
button  pushes,  interpretation)  to  retrieve.  Head  and  eye  tracker  data  are  used  in  combination  with 
BWRS  and  SA  data  to  help  determine  high- workload  tasks.  The  inclusion  of  head  and  eye 
tracker  data  provides  objective  performance  data  that  can  augment  the  subjective  data  collected 
from  the  crew  workload  and  SA  ratings.  When  pilots  spend  an  excessive  amount  of  time  “heads 
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down”  or  inside  the  cockpit,  their  workload  ratings  are  often  higher.  Situational  awareness 
ratings  during  excessive  heads-down  time  are  usually  task-dependent.  In  some  cases,  higher 
workload  may  actually  result  in  higher  SA  if  a  pilot  is  engaging  in  tasks  that  would  increase  SA 
of  a  particular  event  such  as  locating  enemies  or  identifying  terrain  features.  Figure  5  shows  an 
example  of  eye  tracker  data  collected  from  a  copilot-gunner  during  an  AH-64D  simulation. 
ARL/HRED  compiled  a  summary  of  Army  Aviation  eye  tracker  data  (Hicks  et  al.,  2012)  that 
documented  data  collection  methodology  and  results. 


Out-The-Window 

6% 


TEDAC 

14% 


Keyboard 

2% 


MPD 

19% 


Kneeboard/ 
Printed  Map 
6% 


Other 

4% 


Figure  5.  Eye  tracker  data  collected  from  AH-64D  Apache. 


3.  Workload  and  SA  Ratings 


3.1  Bedford  Workload  Rating  Scale  Ratings 

The  pilots  completed  the  BWRS  (appendix  A)  immediately  after  each  mission  to  rate  the  level  of 
workload  that  they  experience  when  performing  flight  and  mission  tasks.  The  ratings  are 
compared  against  the  workload  rating  requirements  for  the  aircraft  (as  applicable)  to  determine  if 
the  crewstation  design  is  imposing  excessive  workload  on  the  pilots.  Table  6  shows  a  summary 
of  overall  workload  averages  collected  during  the  simulations  and  operational  tests.  The  ratings 
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indicate  that  the  pilots  and  copilots  typically  experienced  moderate  overall  workload  with  either 
no  reduction  in  spare  workload  capacity  or  insufficient  to  reduced  spare  workload  capacity 
during  the  simulations  and  tests.  A  rating  of  4  or  higher  on  the  BWRS  indicates  that  the  pilots 
and  copilots  had  a  reduction  in  spare  workload  capacity  to  perform  tasks.  The  one  exception  was 
the  ARH-70  LUT.  Copilots  experienced  high  workload  during  the  LUT  because  the  crewstation 
display  menu  structure  was  nonintuitive  and  time-consuming  to  navigate,  and  the  aircraft  day  TV 
and  Forward-Looking  Infrared  (FLIR)  sensor  was  difficult  to  control.  In  the  case  of  the  3-D 
conformal  tests  for  CH-47F  and  UH-60L/M,  higher  workload  ratings  were  due  to  the  crews 
performing  challenging  tasks  such  as  takeoffs  and  landings  in  brown-out  conditions  using  new 
symbology  designs. 


Table  6.  Overall  workload  averages. 


Bedford  Workload  Ratings  -  Overall  Workload  Averages 

System/Test 

Copilot 

Pilot 

AH-64D  -  Federated  (UAS) 

3.30 

2.60 

AH-64D  -  Integrated  (UAS) 

2.60 

2.90 

AH-64E  IOT 

2.50 

2.50 

AH-64E  LUT 

4.22 

3.22 

ARH  -  CAAS 

3.71 

3.94 

ARH  LUT 

7.39 

4.11 

CH-47F  -  CAAS 

2.66 

2.70 

CH-47F  -  HSDH 

2.41 

3.36 

CH-47F  -  2-D  HUD 

3.31 

3.81 

CH-47F  -  3-D  Conformal 

3.70 

4.10 

C-27J  MOT 

2.17 

2.13 

OH-58F  -  HFE  #2 

3.17 

3.00 

OH-58F  -  HFE  #3 

3.44 

2.56 

RAH-66  -  FDTE  1 

3.08 

2.90 

UH-60M  -  LEUE 

3.33 

2.98 

14 


Table  6.  Overall  workload  averages  (continued). 


Bedford  Workload  Ratings  -  Overall  Workload  Averages 

System/Test 

Copilot 

Pilot 

UH-60M  -  LUT 

2.80 

2.58 

UH-60M  -  EUD 

2.71 

2.16 

UH-60M  -  TAIS  1 

1.87 

2.42 

UH-60M  -  TAIS  2 

2.93 

2.66 

UH-60M  -  MIM  1 

2.22 

2.39 

UH-60M  -  MMI  2 

2.35 

2.20 

UH-60L  -  Multi-Modal  Cues 

2.93 

3.38 

UH-60L  -  3-D  Conformal 

4.27 

4.92 

3.2  SART  and  CLSA  Ratings 

The  pilots  completed  the  SART  or  CLSA  immediately  after  each  mission  to  rate  the  level  of  SA 
they  experienced  while  performing  the  mission.  They  also  rated  their  level  (high-low)  of  SA  of 
battlefield  elements  (e.g.,  location  of  enemy  units).  These  data  provide  ARL  researchers 
information  on  how  well  the  pilots  perceived  the  simulation  environment  and  potential  threats. 
The  battlefield  elements  situation  awareness  questionnaire  is  completed  in  conjunction  with  the 
SART  or  CLSA  questionnaire  after  each  mission.  Table  7  shows  a  summary  of  overall  SART 
and  CLSA  score  averages  collected  during  the  simulations  and  operational  tests.  The  data 
indicate  that  the  pilots  typically  experienced  moderate  to  above  moderate  levels  of  SA.  The  one 
exception  was  the  ARH  CAAS.  The  pilot  and  copilot  SA  ratings  indicate  that  they  experienced 
lower  levels  of  SA  due  to  the  workload  required  to  navigate  the  display  menu  structure  and 
difficulty  controlling  the  aircraft  day  TV  and  FLIR.  The  CLSA  data  for  the  UH-60L  and  CH-47F 
were  obtained  using  an  inverse  scale  where  higher  ratings  were  indicative  of  better  S  A  as 
opposed  to  the  traditional  CLSA  scale.  CLSA  ratings  would  have  ranged  between  1  and  2  for  the 
UH-60L  and  CH-47F  if  they  had  been  obtained  with  the  traditional  scale.  SA  ratings  for  the 
AH-64E  IOT,  ARH  LUT,  and  C-27J  MOT  were  not  available  for  inclusion  in  the  table. 
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Table  7.  Overall  SART  and  CLSA  averages. 


Situation  Awareness  Rating  Technique  -  Overall  Averages 

System/Test 

Copilot 

Pilot 

AH-64D  -  Federated  (UAS) 

19.00 

21.30 

AH-64D  -  Integrated  (UAS) 

18.40 

23.20 

ARH  -  CAAS 

17.67 

17.22 

CH-47F  -  CAAS 

23.83 

20.13 

RAH-66  -  FDTE  1 

21.86 

22.40 

UH-60M  -  LEUE 

26.42 

25.25 

UH-60M  -  LUT 

28.28 

28.22 

UH-60M  -  TAIS  1 

33.78 

18.66 

UH-60M  -  TAIS  2 

17.66 

20.60 

UH-60L  -  Multi-Modal  Cues 

29.33 

19.95 

China  Lake  Situational  Awareness  Scale  -  Overall  Averages 

System/Test 

Co-Pilot 

Pilot 

AH-64E  LUT 

1.94 

1.74 

OH-58F  -  HFE  #2 

2.75 

2.50 

OH-58F  -  HFE  #3 

2.22 

1.67 

UH-60L  -  Multi-Modal  Cues 

3.94 

3.75 

UH-60L  -  3-D  Conformal 

3.74 

3.67 

CH-47F  -  3-D  Conformal 

3.91 

3.74 

4.  Summary 


The  method  that  ARL/HRED  uses  to  assess  the  human  factors  characteristics  of  U.S.  Army 
Aviation  helicopter  crewstations  has  been  successful  in  evaluating  crew  workload  and  SA,  and 
eliminating  human  factors  design  problems.  To  date,  more  than  500  crewstation  design 
improvements  have  been  made  to  resolve  the  human  factors  problems  and  enhance  crew 
performance.  Examples  include  software  improvements  to  crewstation  displays  such  as  enhanced 
functionality  and  presentation  of  display  pages  to  pilots,  improved  color-coding  of  battlefield 
graphics,  reduced  number  of  button  presses  to  display  information,  enhanced  readability  of 
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display  map  pages,  and  improved  presentation  of  aircraft  operational  limits.  Hardware 
improvements  include:  modifications  to  crewstation  seats,  consoles,  and  glare  shields  to  improve 
visual  access  and  physical  reach  to  displays  and  controls;  improved  functionality  of  flight  helmets 
and  helmet-mounted  displays;  and  optimized  crewstation  switch  location  and  function.  These 
improvements  are  documented  in  ARL  technical  reports  and  Army  Aviation  programmatic  and 
technical  documents. 

The  benefits  to  using  the  crewstation  assessment  method  to  assess  crew  workload,  SA,  and 
crewstation  design  are  (1)  iterative  crewstation  evaluations  drive  continuous  incremental 
improvements,  (2)  improvements  are  identified  in  near  real  time,  which  aids  rapid  modification, 
(3)  identifies  crewstation  design  that  needs  further  improvement,  (4)  issues  documented  for  one 
aircraft  often  apply  to  new  or  updated  aircraft,  which  helps  with  early  identification  of  issues  for 
these  aircraft,  and  (5)  results  feed  the  assessments  used  by  acquisition  officials  to  determine 
whether  to  manufacture  and  field  Army  Aviation  aircraft. 

ARL/HRED  will  continue  to  assess  crew  workload  and  SA  and  thereby  improve  crewstation 
design  to  meet  the  demands  of  the  next  generation  of  Army  aircraft. 
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Appendix  A.  Bedford  Workload  Rating  Scale  and  Questionnaire 


The  appendix  appears  in  its  original  form,  without  editorial  change. 
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Workload 


Rate  the  workload  for  the  Flight  and  Mission  Tasks  you  performed  (on  the  2nd  page)  using  the 
workload  scale  below.  Place  the  workload  rating  in  the  blank  next  to  each  Flight  and  Mission 
Task. 


Workload  Description 


YES 


Is  workload 
satisfactory 
without  reduction  in 
spare  (workload)  capacity? 


NO 


Is  workload  tolerable 
for  the  task? 


NO 


‘Rating’ 


Workload  insignificant 

1 

Workload  low 

2 

Enough  spare  capacity  for  all 
desirable  additional  tasks 

3 

Is  it  possible  to 
complete  the  task? 


Pilot  Decisions 


Insufficient  spare  capacity  for  easy 
attention  to  additional  tasks 

4 

Reduced  spare  capacity.  Additional 
tasks  cannot  be  given  the  desired 
amount  of  attention 

5 

Little  spare  capacity:  level  of  effort 
allows  little  attention  to  additional 
tasks 

6 

Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 

7 

Very  high  workload  with  almost  no 
spare  capacity.  Difficulty  in 
maintaining  level  of  effort 

8 

Extremely  high  workload.  No  spare 
capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 

9 

Task  abandoned.  Pilot  unable  to 

10 

apply  sufficient  effort 
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Pilot  Workload 


Rate  the  workload  for  the  Flight  and  Mission  Tasks  you  performed  during  the  mission  that  you 
just  completed.  Use  the  scale  provided  on  the  last  page  of  this  questionnaire.  Place  the  workload 
rating  in  the  blank  next  to  each  Flight  and  Mission  Task.  If  you  did  not  perform  a  task  during  the 
mission  that  you  just  completed,  place  an  X  in  the  non-applicable  (NA)  column. 


Task 

No. 

Flight  and  Mission  Tasks 

Workload  Rating 

NA 

1026 

Maintain  Airspace  Surveillance 

1028 

Perform  Hover  Power  Check 

1030 

Perform  Hover  Out-Of-Ground-Effect  (OGE)  Check 

1032 

Perform  Radio  Communication  Procedures 

1038 

Perform  Hovering  Flight 

1040 

Perform  Visual  Meteorological  Conditions  (VMC)  Takeoff 

1044 

Navigate  by  Pilotage  and  Dead  Reckoning 

1046 

Perform  Electronically  Aided  Navigation 

1048 

Perform  Fuel  Management  Procedures 

1052 

Perform  VMC  Flight  Maneuvers 

1058 

Perform  VMC  Approach 

.... 

Level  of  Interoperability  (LOI)  2  with  UAS 

1066 

Perform  A  Running  Landing 

1070 

Respond  to  Emergencies 

1074 

Respond  to  Engine  Failure  in  Cruise  Flight 

1140 

Perform  Nose  Mounted  Sensor  (NMS)  Operations 

1142 

Perform  Digital  Communications 

1155 

Negotiate  Wire  Obstacles 

1170 

Perform  Instrument  Takeoff 

1176 

Perform  Non  Precision  Approach  (GCA) 

1178 

Perform  Precision  Approach  (GCA) 

1180 

Perform  Emergency  GPS  Recovery  Procedure 

1082 

Perform  an  Autorotation 

1182 

Perform  Unusual  Attitude  Recovery 

1188 

Operate  AS  E/transponder 

1184 

Respond  to  IMC  Conditions 

1194 

Perform  Refueling  /Rearming  Operations 

1404 

Perform  Electronic  Countermeasures  ElectronicVCounter- 
Countermeasures 

1405 

Transmit  Tactical  Reports 

1407 

Perform  Terrain  Flight  Takeoff 

1408 

Perform  Terrain  Flight 

1409 

Perform  Terrain  Flight  Approach 

1410 

Perform  Masking  and  Unmasking 

1411 

Perform  Terrain  Flight  Deceleration 
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Task 

No. 

Flight  and  Mission  Tasks 

Workload  Rating 

NA 

1413 

Perform  Actions  on  Contact 

1416 

Perform  Weapons  Initialization  Procedures 

1422 

Perform  Firing  Techniques 

1456 

Engage  Target  with  .50  Cal 

1458 

Engage  Target  with  Hellfire 

1462 

Engage  Target  with  Rockets 

1472 

Perform  Aerial  Observation 

1471 

Perform  Target  Handover 

1472 

Aerial  Observation 

1473 

Call  for  Indirect  Fire 

2010 

Perform  Multi -Aircraft  Operations 

2127 

Perform  Combat  Maneuvering  Flight 

2128 

Perform  Close  Combat  Attack 

2129 

Perform  Combat  Position  Operations 

2164 

Call  for  Tactical  Air  Strike 

Zone  Reconnaissance 

Route  Reconnaissance 

Area  Reconnaissance 

Aerial  Surveillance 

Overall  Workload  for  the  Mission 

If  you  gave  a  workload  rating  of  ‘5’  or  higher  for  any  task,  explain  why  the  workload  was  high 
for  the  task. 
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Appendix  B.  SART  and  CLSA  Rating  Scales 


The  appendix  appears  in  its  original  form,  without  editorial  change. 
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Situation  Awareness  is  defined  as  “timely  knowledge  of  what  is  happening  as  you  perform 
your  right  or  left  seat  tasks  during  the  mission.” 


Situation  Awareness  Rating  Technique  (SART) 

DEMAND 

Instability  of  Situation 

Likeliness  of  situation  to  change  suddenly 

Variability  of  Situation 

Number  of  variables  which  require  your  attention 

Complexity  of  Situation 

Degree  of  complication  (number  of  closely  connected  parts)  of  the  situation 

SUPPLY 

Arousal 

Degree  to  which  you  are  ready  for  activity 

Spare  Mental  Capacity 

Amount  of  mental  ability  available  to  apply  to  new  tasks 

Concentration 

Degree  to  which  your  thoughts  are  brought  to  bear  on  the  situation 

Division  of  Attention 

Amount  of  division  of  your  attention  in  the  situation 

UNDERSTANDING 

Information  Quantity 

Amount  of  knowledge  received  and  understood 

Information  Quality 

Degree  of  goodness  or  value  of  knowledge  communicated 

Familiarity 

Degree  of  acquaintance  with  the  situation 
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Rate  the  level  of  each  component  of  situation  awareness  that  you  had  when  you  performed 
‘flying  pilot’  tasks  in  the  right  seat  -or-  ‘non-flying’  pilot  tasks  in  the  left  seat  during  the 
mission  that  you  just  completed.  Circle  the  appropriate  number  for  each  component  of  situation 
awareness  (e.g.,  complexity  of  situation). 

DEMAND 

Instability  of  situation:  Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Variability  of  situation:  Low  1 - 2 - 3 . —4 - 5 - 6 - 7  High 

Complexity  of  situation:  Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 


Arousal: 

Low 

Spare  mental  capacity: 

Low 

Concentration: 

Low 

Division  of  attention: 

Low 

SUPPLY 

1 - 2 - 3- . -4— . 5 . -6- . 7  High 

1 - 2 - 3 - 4- . —5 - 6 - 7  High 

1 - 2 - 3- . -4 - 5- . -6- . -7  High 

1— . -2 - 3 - 4 - 5 - 6 - 7  High 


UNDERSTANDING 

Information  quantity: 

Information  quality: 

Lamiliarity: 

Battlefield  Elements 

Rate  the  level  of  situation  awareness  you  had  for  each  battlefield  element  during  the  mission? 
(Place  an  X  in  the  appropriate  column  for  each  battlefield  element). 


Low  1 - 2 - 3 - 4 - 5 - 6— . -7  High 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 
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Battlefield 

Elements 

Very  High 
Level  of 
Situation 
Awareness 

Fairly  High 
Level  of 
Situation 
Awareness 

Borderline 

Fairly  Low 
Level  of 
Situation 
Awareness 

Very  Low 
Level  of 
Situation 
Awareness 

Location  of 
Enemy  Units 

Location  of 
Friendly  Units 

Location  of 
Non- 

Combatants 
(e.g..  Civilians) 

Location  of  My 
Aircraft  During 
Missions 

Location  of 
Other  Aircraft 

In  My  Flight 

Location  of 
Cultural 
Features  (e.g., 
bridges) 

Route 

Information 
(ACPs,  BPs, 
EAs,  RPs,  etc.) 

Status  of  My 
Aircraft  Systems 
(e.g.,  fuel 
consumption) 

Describe  any  instances  when  you  had  low  situation  awareness  during  the  mission: 


China  Lake  Situational  Awareness  Scale 
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CLSA  SCALE  VALUE 

CONTENT 

Very  Good  -  1 

•  Full  knowledge  of  A/C  energy  state/tactical  environment/mission. 

•  Full  ability  to  anticipate/accommodate  trends. 

Good  -  2 

•  Full  knowledge  of  A/C  energy  state/tactical  environment/mission. 

•  Partial  ability  to  anticipate/accommodate  trends; 

•  No  task  shedding 

Adequate  -  3 

•  Full  knowledge  of  A/C  energy  state/tactical  environment/mission. 

•  Saturated  ability  to  anticipate/accommodate  trends; 

•  Some  shedding  of  minor  tasks 

Poor  -  4 

•  Fair  knowledge  of  A/C  energy  state/tactical  environment/mission. 

•  Saturated  ability  to  anticipate/accommodate  trends; 

•  Shedding  of  all  minor  tasks  as  well  as  many  not  essential  to  flight 
safety/mission  effectiveness 

Very  Poor  -  5 

•  Minimal  knowledge  of  A/C  energy  state/tactical  environment/mission. 

•  Oversaturated  ability  to  anticipate/accommodate  trends; 

•  Shedding  of  all  tasks  not  essential  to  flight  safety/mission  effectiveness 
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Intentionally  left  blank. 
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Appendix  C.  PCI  Questionnaire  Example 


The  appendix  appears  in  its  original  form,  without  editorial  change. 
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PV1.  The  following  table  lists  the  components  of  a  CAAS  crewstation.  For  each  component, 
indicate  whether  or  not  you  experienced  a  problem  using  the  component  in  a  quick  and  efficient 
manner  during  the  mission  you  just  completed.  Check  ‘Yes’  if  you  experienced  one  or  more 
problems.  Check  ‘No’  if  you  did  not  experience  any  problems.  Check  ‘Not  Used’  if  you  did  not 
use  the  component  during  the  mission  you  just  completed. 

•  Multifunction  Displays  (MFD) 


o  Vertical  Situation  Display  (VSD) 

Yes 

No 

Not  Used. 

o  VSD  Hover  (VSDH) 

Yes 

No 

Not  Used. 

o  Horizontal  Situation  Display  (HSD) 

Yes 

No 

Not  Used. 

o  HSD  Hover  (HSDH) 

Yes 

No 

Not  Used. 

o  EOS 

Yes 

No 

Not  Used. 

o  Digital  Map  Display  (DMS) 

Yes 

No 

Not  Used. 

o  Warning,  Caution,  Advisory 

Display  (WCA) 

Yes 

No 

Not  Used. 

o  Engine  Instrument  Caution 

Advisory  System  (EICAS) 

Yes 

No 

Not  Used. 

Control  Display  Unit  (CDU) 

o  Initializing  CDU 

Yes 

No 

Not  Used 

o  Managing  GPS  /  Flight  Plan 

Yes 

No 

Not  Used 

o  Managing  COM,  NAY,  IFF  (CNI) 

Yes 

No 

Not  Used. 

If  you  answered  “Yes”  to  any  of  the  questions,  describe  a)  the  problems  you  experienced,  b)  how 
much  the  problems  degraded  your  performance,  and  c)  any  recommendation  you  have  for 
improving  the  design  of  the  various  functional  components. 


PY3.  Please  answer  the  following  questions  regarding  the  Multifunction  Control  Unit  (MFCU): 

PY3-1.  Did  the  functionality  of  the  directional  control  and  switches  on  the  MFCU  perform  the 
actions  you  expected? 
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Yes _  No _  Not  Used _ 

PY3-2.  Was  the  sensitivity  of  the  directional  control  appropriate? 

Yes _  No _  Not  Used _ 

PY3-3.  Did  you  experience  abnormal  hand  discomfort  while  using  the  MFCU? 

Yes _  No _  Not  Used _ 

PV3-4.  Did  you  have  adequate  space  in  the  cockpit  to  use  the  MFCU? 

Yes _  No _  Not  Used _ 

If  you  experienced  any  problems  with  the  MFCU,  please  describe  the  problems  in  as  much  detail 

as  you  can  and  make  recommendations  to  correct  the  problems. 


PV4.  Did  you  have  difficulty  using  any  of  the  switches  on  the  collective  or  cyclic  grips? 

Collective  Grip  Yes _  No _ 

Cyclic  Grip  Yes _  No _ 

If  you  answered  “Yes”  for  either  flight  control,  please  list  which  flight  control  and  switch(es), 
and  the  problems  you  experienced  (e.g.,  confused  two  switches  due  to  similar  shape,  switch  too 
hard  to  reach). 


PV5.  Was  there  any  symbology  depicted  on  the  following  displays/pages  that  was  difficult  to 
quickly  and  easily  understand,  cluttered,  or  otherwise  difficult  to  use? 


Vertical  Situation  Display  (VSD) 

Yes 

No  _ 

VSD  Hover  (VSDH) 

Yes 

No  _ 

Horizontal  Situation  Display  (HSD) 

Yes 

No  _ 

HSD  Hover  (HSDH) 

Yes 

No  _ 

EICAS 

Yes 

No  _ 

Digital  Map  System  (DMS) 

Yes 

No  _ 

31 


If  you  answered  “Yes”  to  any  of  the  questions,  please  describe  a)  the  display/page,  b)  the 
symbology  that  was  difficult  to  understand,  c)  how  the  symbology  may  have  degraded  your 
performance,  and  d)  any  recommendations  you  have  for  improving  the  design  of  the  various 
functional  components. 


PY6.  How  would  you  rate  your  ability  to  detect  the  following  occurrences  based  on  the 


characteristics  of  the  flight  displays? 

Caution/Advisory  (MFD) 

1 

2 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Warning  (Master  Warning  Panel) 

1 

2 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Entry  Into  Operational  Limits 

1 

2 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Low  Luel  (MED) 

1 

2 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 
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If  you  answered  “Somewhat  Difficult”,  or  “Very  Difficult”,  please  indicate  which  annunciation 
you  had  difficulty  detecting,  why  you  may  have  had  difficulty  detecting  it,  and  any 
recommendations  to  make  the  annunciation  more  easily  detectable. 


PV7.  Based  on  the  missions  you’ve  conducted  this  week,  what  are  the  top  enhancements  that 
should  be  made  to  the  crewstation  to  improve  pilot  performance? 
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Intentionally  lelt  blank. 
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Appendix  D.  SME  Questionnaire 


The  appendix  appears  in  its  original  form,  without  editorial  change. 
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Workload 


Was  workload  tolerable 
for  the  mission  tasks? 


i 


Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 


Very  high  workload  with  almost  no 
spare  capacity.  Difficulty  in 
maintaining  level  of  effort 


Extremely  high  workload.  No  spare 
capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 


36 


TSCWL1.  Place  the  workload  rating  in  the  blank  next  to  each  crewmember  using  the  rating  scale 
on  the  previous  page  (continued). 


Crewmembers 

Overall  Workload  Rating  For  This 
Mission 

Left  Seat 

Right  Seat 

If  you  assigned  a  workload  rating  of  ‘6’  or  higher  for  either  crewmember,  explain  why: 


TSCWL2.  Which  crewmember  was  the  ‘flying  pilot’  for  most  of  the  mission? 

Left  seat  _  Right  seat  _ 

TSCWL3.  What  percentage  of  the  time  was  the  crewmember  (left  seat  or  right  seat  in  question 
above)  the  ‘flying  pilot’  during  the  mission? 

_ % 

TSCWL4.  Rate  the  effectiveness  of  aircrew  coordination  as  defined  by  the  USAAVNC  Aircrew 
Coordination  ETP  and  TC  1-210. 

1  2  3  4  5 


Excellent  Good 


Average  Needs  Improvement  Unacceptable 
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Situation  Awareness 


Rating 

Check  One 

Crew  was  consistently  aware  of  all  entities  on  the  battlefield 

Crew  was  aware  of  the  battlefield  with  minor  or  insignificant  variation  between  perception  and 

reality. 

Crew  was  aware  of  the  battlefield.  Variation  between  reality  and  perception  did  not 
significantly  impact  mission  success. 

SA  needs  improvement.  Lack  of  SA  had  some  negative  effect  on  the  success  of  the  mission. 

Lack  of  SA  caused  mission  failure. 

Describe  any  problems  that  aircrews  had  with  situation  awareness. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 

ARH  Armed  Reconnaissance  Helicopter 

ARL  U.S.  Army  Research  Laboratory 

BHEAC  Blackhawk  Helicopter  Engineering  and  Analysis  Cockpit 

BHIVE  Battlefield  Highly  Immersive  Virtual  Environment 

BWRS  Bedford  Workload  Rating  Scale 

CAAS  Common  Aviation  Architecture  System 

CHEAC  Cargo  Helicopter  -  Engineering  Analysis  Cockpit 

CPC  Comanche  Portable  Cockpit 

EDS  Engineering  Development  Simulator 

EUD  Early  User  Demonstration 

FDT&E  Force  Development  Test  and  Evaluation 

FLIR  Forward-Looking  Infrared 

HFE  Human  Factors  Engineering 

HRED  Human  Research  and  Engineering  Directorate 

IOT  Initial  Operational  Test 

LEUE  Limited  Early  User  Evaluation 

LUT  Limited  User  Test 

MIM  Mission  Information  Management 

MOT  Multi-Service  Operational  Test 

PCI  Pilot-Crewstation  Interface 

RACRS  Risk  and  Cost  Reduction  System 

RNAV  Area  Navigation 

RNP  Required  Navigation  Performance 

SA  situation  awareness 

SART  Situation  Awareness  Rating  Technique 

SIL  System  Integration  Laboratory 


39 


SME 

TAIS 

UAS 


subject  matter  expert 

Tactical  Airspace  Integration  System 

Unmanned  Aircraft  System 
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